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Abstract 

Background: Two of tine current metlnodological barriers to implementation science efforts are the lack of 
agreement regarding constructs hypothesized to affect implementation success and identifiable measures of these 
constructs. In order to address these gaps, the main goals of this paper were to identify a multi-level framework 
that captures the predominant factors that impact implementation outcomes, conduct a systematic review of 
available measures assessing constructs subsumed within these primary factors, and determine the criterion validity 
of these measures in the search articles. 

Method: We conducted a systematic literature review to identify articles reporting the use or development of 
measures designed to assess constructs that predict the implementation of evidence-based health innovations. 
Articles published through 12 August 2012 were identified through MEDLINE, CINAHL, PsyclNFO and the journal 
Implementation Science. We then utilized a modified five-factor framework in order to code whether each measure 
contained items that assess constructs representing structural, organizational, provider, patient, and innovation level 
factors. Further, we coded the criterion validity of each measure within the search articles obtained. 

Results: Our review identified 62 measures. Results indicate that organization, provider, and innovation-level 
constructs have the greatest number of measures available for use, whereas structural and patient-level constructs 
have the least. Additionally, relatively few measures demonstrated criterion validity, or reliable association with an 
implementation outcome {e.g., fidelity). 

Discussion: In light of these findings, our discussion centers on strategies that researchers can utilize in order to 
identify, adapt, and improve extant measures for use in their own implementation research. In total, our literature 
review and resulting measures compendium increases the capacity of researchers to conceptualize and measure 
implementation-related constructs in their ongoing and future research. 

Keywords: Implementation, Health innovation. Evidence-based practice. Systematic review. Measure, Questionnaire, 
Scale 
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Background 

Each year, billions of dollars are spent in countries around 
the world to support the development of evidence-based 
health innovations [1,2] — interventions, practices, and 
guidelines designed to improve human health. Yet, only 
a small fraction of these innovations are ever implemented 
into practice [3], and efforts to implement these practices 
can take many years [4]. Thus, new approaches are greatly 
needed in order to accelerate the rate at which existing 
and emergent knowledge can be implemented in health- 
related settings around the world. 

As a number of scholars have noted, researchers currently 
face significant challenges in measuring implementation- 
related phenomena [5-7]. The implementation of evidence- 
based health innovations is a complex process. It involves 
attention to a wide array of multi-level variables related to 
the innovation itself, the local implementation context, and 
the behavioral strategies used to implement the innovation 
[8,9]. In essence, there are many 'moving parts' to consider 
that can ultimately determine whether implementation 
efforts succeed or fail. 

These challenges also stem from heterogeneity across 
the theories and frameworks that guide implementation 
research. There is currently no single theory or set of 
theories that offer testable hypotheses about when and 
why specific constructs will predict specific outcomes 
within implementation science [5,10]. What does exist 
in implementation science, however, are a plethora of 
frameworks that identify general classes or typologies of 
factors that are hypothesized to affect implementation 
outcomes (i.e., impact frameworks [5]). Further, within 
the available frameworks, there is considerable hetero- 
geneity in the operationalization of constructs of interest 
and the measures available to assess them. At present, 
constructs that have been hypothesized to affect imple- 
mentation outcomes are often poorly defined within 
studies [11,12]. And, the measures used to assess these 
constructs are frequently developed without direct con- 
nection to substantive theory or guiding frameworks and 
with minimal analysis of psychometric properties, such 
as internal reliability and construct validity [12]. 

In light of these measurement-related challenges, in- 
creasing the capacity of researchers to both conceptualize 
and measure constructs hypothesized to affect implemen- 
tation outcomes is a critical way to advance the field of 
implementation science. With these limitations in mind, 
the main goals of the current paper were threefold. First, 
we expanded existing multi-level frameworks in order 
to identify a five-factor framework that organizes the 
constructs hypothesized to affect implementation outcomes. 
Second, we conducted a systematic review in order to 
identify measures available to assess constructs that can 
conceivably act as causal predictors of implementation 
outcomes, and coded whether each measure assessed 



any of the five factors of the aforementioned framework. 
And third, we ascertained the criterion validity — whether 
each measure is a reliable predictor of implementation 
outcomes (e.g., adoption, fidelity) — of these measures iden- 
tified in the search articles. 

A multi-level framework guiding Implementation science 
research 

Historically, there has been great heterogeneity in the focus 
of implementation science frameworks. Some frameworks 
examine the impact of a single type of factor, positing that 
constructs related to the individual provider (e.g., practi- 
tioner behavior change: Transtheoretical Model [13,14]) or 
constructs related to the organization (e.g., organizational 
climate for implementation: Implementation Effectiveness 
model [15]) impact implementation outcomes. More 
recently, however, many frameworks have converged to 
outline a set of multi-level factors that are hypothesized 
to impact implementation outcomes [9,16-18]. These 
frameworks propose that implementation outcomes are 
a function of multiple types of broad factors that can 
be hierarchically organized to represent micro-, meso-, 
and macro-level factors. 

What, then, are the multi-level factors hypothesized to 
affect the successful implementation of evidence-based 
health innovations? In order to address this question, 
Durlak and DuPre [19] reviewed meta-analyses and add- 
itional quantitative reports examining the predictors of 
successful implementation from over 500 studies. In 
contrast, Damschroder et al. [20] reviewed 19 existing 
implementation theories and frameworks in order to 
identify common constructs that affect successful imple- 
mentation across a wide variety of settings (e.g., healthcare, 
mental health services, corporations). Their synthesis yielded 
a typology (i.e., the Consolidated Framework for Implemen- 
tation Research [CFIR]) that largely overlaps with Durlak 
and DuPre's [19] analysis. Thus, although these researchers 
utilized different approaches — with one identifying uni- 
fying constructs from empirical results [20] and the 
other identifying unifying constructs from existing concep- 
tual frameworks [19] — both concluded that structural- (i.e., 
community-level [19]; outer-setting [20]), organizational- 
(i.e., prevention delivery system organizational capacity 
[19]; inner setting [20]), provider-, and innovation-level 
factors predict implementation outcomes [19,20]. 

The structural-level factor encompasses a number of 
constructs that represent the outer setting or external 
structure of the broader sociocultural context or com- 
munity in which a specific organization is nested [3]. 
These constructs could represent aspects of the physical 
environment (e.g., topographical elements that pose barriers 
to clinic access), political or social climate (e.g., liberal ver- 
sus conservative), public policies (e.g., presence of state 
laws that criminalize HIV disclosure), economic climate 
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{e.g., reliance upon and stability of private, state, federal 
funding), or infrastructure {e.g., local workforce, quality 
of public transportation surrounding implementation 
site) [21,22]. 

The organizational-level factor encompasses a number 
of constructs that represent aspects of the organization 
in which an innovation is being implemented. These 
aspects could include leadership effectiveness, culture or 
climate {e.g., innovation climate, the extent to which 
organization values and rewards evidence-based practice 
or innovation [23]), and employee morale or satisfaction. 

The provider-level factor encompasses a number of 
constructs that represent aspects of the individual pro- 
vider who implements the innovation with a patient or 
client. We use 'provider' as an omnibus term that refers 
to anyone who has contact with patients for the purposes 
of implementing the innovation, including physicians, other 
clinicians {e.g., psychologists), allied health professionals 
{e.g., dieticians), or staff {e.g., nurse care managers). 
These aspects could include attitudes towards evidence- 
based practice [24] or perceived behavioral control for 
implementing the innovation [25]. 

The innovation-level factor encompasses a number of 
constructs that represent aspects of the innovation that 
will be implemented. These aspects could include the 
relative advantage of utilizing an innovation above existing 
practices [26] and quality of evidence supporting the 
innovation's efficacy (Organization Readiness to Change 
Assessment, or ORCA [27]). 

But, where does the patient or client fit in these 
accounts? The patient-level factor encompasses patient 
characteristics such as health-relevant beliefs, motivation, 
and personality traits that can impact implementation 
outcomes [28] . In efficacy trials that compare health 
innovations to a standard of care or control condition, 
patient-level variables are of primary importance both as 
outcome measures of efficacy {e.g., improved patient health 
outcomes) and as predictors {e.g., patient health literacy, 
beliefs about innovation success) of these efficacy outcomes. 
Patient-level variables such as behavioral risk factors {e.g., 
alcohol use [29]) and motivation [30,31] often moderate 
the retention in and efficacy of behavioral risk reduction 
interventions. Moreover, patients' distrust of medical 
practices and endorsement of conspiracy beliefs have 
been linked to poorer health outcomes and retention in 
care, especially among African-American patients and 
other vulnerable populations [32]. However, in imple- 
mentation trials testing whether and to what degree an 
innovation has been integrated into a new delivery con- 
text, the outcomes of interest are different from those 
in efficacy trials, typically focusing on provider- or 
organizational-level variables [33,34]. 

Despite the fact that they focus on different outcomes, 
what implementation trials have in common with efficacy 



trials is that patient-level variables are important to 
examine as predictors, because they inevitably impact 
the outcomes of implementation efforts [28,35]. The 
very conceptualization of some implementation outcomes 
directly implicates the involvement of patients. For ex- 
ample, fidelity, or 'the degree to which an intervention 
was implemented as it was prescribed in the original 
protocol or as it was intended by the program developers' 
[6], necessarily involves and is affected by patient- level 
factors. Further, as key stakeholders in all implementation 
efforts, patients are active agents and consumers of 
healthcare from whom buy-in is necessary. In fact, in 
community-based participatory research designs, patients 
are involved directly as partners in the research process 
[36,37]. Thus, as these findings reiterate, patient- level 
predictors explain meaningful variance in implementation 
outcomes, making failure to measure these variables as 
much a statistical as a conceptual omission. 

For the aforementioned reasons, we posit that a com- 
prehensive multi-level framework must include a patient- 
level factor. Therefore, in the current review, we employed 
a comprehensive multi-level framework positing five factors 
representing structural-, organizational-, patient-, provider-, 
and innovation-levels of analysis. We utilized this five- 
factor framework as a means of organizing and describing 
important sets of constructs, as well as the measures 
that assess these constructs. Figure 1 depicts our current 
conceptual framework. The left side of the figure depicts 
causal factors, or the structural-, organizational-, patient-, 
provider-, and innovation-level constructs that are hypoth- 
esized to cause or predict implementation outcomes. 
These factors represent multiple levels of analysis, from 
micro-level to macro-level, such that a specific innovation 
{e.g., evidence-based guideline) is implemented by providers 
to patients who are nested within an organization {e.g., 
clinical care settings), which is nested within a broader 
structural context {e.g., healthcare system, social climate, 
community norms). The right side of the figure depicts 
the implementation outcomes — such as adoption, fidelity, 
implementation cost, penetration, and sustainability [6] — 
that are affected by the causal factors. Together, these 
factors illustrate a hypothesized causal effect wherein 
constructs lead to implementation outcomes. 

Available measures 

What measures are currendy available to assess constructs 
within these five factors hypothesized to predict implemen- 
tation outcomes? The current systematic review seeks to 
answer this basic question and act as a guide to assist 
researchers in identifying and evaluating the types of 
measures that are available to assess structural, orga- 
nizational, provider, patient, and innovation-level constructs 
in implementation research. 
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Figure 1 A multi-level framework predicting implementation outcomes. 
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A number of researchers have also provided reviews 
of limited portions of this literature [38-41]. For example, 
French et al. [40] focused on the organizational-level by 
conducting a systematic review to identify measures 
designed to assess features of the organizational context. 
They evaluated 30 measures derived from both the health- 
care and management/organizational science literatures, 
and their review found support for the representation of 
seven primary attributes of organizational context across 
available measures: learning culture, vision, leadership, 
knowledge need/capture, acquiring new knowledge, 
knowledge sharing, and knowledge use. Other systematic 
reviews and meta-analyses have focused on measures 
that assess provider-level constructs, such as behavioral 
intentions to implement evidence-based practices [38] 
and other research-related variables (e.g., attitudes toward 
and involvement in research activities) and demographic 
attributes {e.g., education [41]). Finally, it is important to 
note that other previous reviews have focused on the 
conceptualization [6] and evaluation [12] of implementa- 
tion outcomes, including the psychometric properties of 
research utilization measures [12]. 

To date, however, no systematic reviews have examined 
measures designed to assess constructs representing the 
five types of factors — structural, organizational, provider, 
patient, and innovation — hypothesized to predict imple- 
mentation outcomes. The purpose of the current system- 
atic review is to identify measures available to assess this 
full range of five factors. In doing so, this review is 
designed to create a resource that will increase the cap- 
acity of and speed with which researchers can identify and 
incorporate these measures into ongoing research. 



Method 

We located article records by searching MEDLINE, 
PsycINFO, and CINAHL databases and abstracts of 
articles published in the journal Implementation Science 
through 12 August 2012. There was no restriction on 
beginning date of this search. (See Additional file 1 for 
full information about the search process). We searched 
with combinations of keywords representing three cat- 
egories: implementation science, health, and measures. 
Given that the field of implementation science includes 
terminology contributions from many diverse fields and 
countries, we utilized thirteen phrases identified as com- 
mon keywords from Rabin et al.'s systematic review of 
the literature [34]: diffusion of innovations, dissemin- 
ation, effectiveness research, implementation, knowledge 
to action, knowledge transfer, knowledge translation, re- 
search to practice, research utilization, research utilisa- 
tion, scale up, technology transfer, translational research. 

As past research has demonstrated, use of database 
field codes or query filters is an efficient and effective 
strategy for identifying high-quality articles for individual 
clinician use [42,43] and systematic reviews [44]. In es- 
sence, these database restrictions can serve to lower the 
number of 'false positive' records identified in the search 
of the literature, creating more efficiency and accuracy 
in the search process. In our search, we utilized several 
such database restrictions in order to identify relevant 
implementation science-related measures. 

In our search of PsycINFO and CINAHL, we used 
database restrictions that allowed us to search each of 
the implementation science keywords within the meth- 
odology sections of records via PsycINFO (i.e., 'tests and 
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measures' field) and CINAHL {i.e., 'instrumentation' field). 
In our hand search of Implementation Science we searched 
for combinations of the keyword 'health' and the imple- 
mentation science keywords in the abstract and title. In 
our search of MEDLINE, we used a database restriction 
that allowed us to search for combinations of the keyword 
'health,' the implementation science keywords, and the 
keywords 'measure,' 'questionnaire,' 'scale,' 'survey,' or 'tool' 
within the abstract and title of records listed as 'validation 
studies.' There were no other restrictions based on study 
characteristics, language, or publication status in our 
search for article records. 

Screening study records and identifying measures 

Article record titles and abstracts were then screened 
and retained for further review if they met two inclusion 
criteria: written in English, and validated or utilized at 
least one measure designed to quantitatively assess a 
construct hypothesized to predict an implementation 
science related outcome {e.g., fidelity, exposure [6,34]). 
Subsequently, retained full-text articles were then 
reviewed and vetted further based on the same two in- 
clusion criteria utilized during screening of the article 
records. The remaining full-text articles were then 
reviewed in order to extract all measures utilized to as- 
sess constructs hypothesized to predict an implementa- 
tion science related outcome. Whenever a measure was 
identified from an article that was not the original valid- 
ation article, we used three methods to obtain full infor- 
mation: ancestry search of the references section of 
article in which the measure was identified, additional 
database and Internet searches, and directly contacting 
corresponding authors via email. 

Measure and criterion validity coding 

We then coded each of the extracted measures to deter- 
mine whether it included items assessing each of the 
five factors — structural, organizational, provider, patient, 
and innovation — based on our operational definitions noted 
above. Items were coded as structural-level factors if they 
assess constructs that represent the structure of the broader 
sociocultural context or community in which a specific 
organization is nested. For example, the Organizational 
Readiness for Change scale [45] assesses several features 
of structural context in which drug treatment centers 
exist, including facility structure (independent versus 
part of a parent organization) and characteristics of the 
service area (rural, suburban, or urban). Items were coded 
as organizational level factors if they assess constructs that 
represent the organization in which an innovation is being 
implemented. For example, the ORCA [27] assesses 
numerous organizational constructs including culture 
(e.^., 'senior leadership in your organization reward clinical 
innovation and creativity to improve patient care') and 



leadership {e.g., 'senior leadership in your organization 
provide effective management for continuous improve- 
ment of patient care'). Items were coded as provider- 
level factors if they assess constructs that represent 
aspects of the individual provider who implements the 
innovation. For example, the Evidence-Based Practice 
Attitudes Scale [24] assesses providers' general attitudes 
towards implementing evidence-based innovations {e.g., 
'I am willing to use new and different types of therapy/ 
interventions developed by researchers') whereas the 
Big 5 personaUty questionnaire assesses general person- 
ality traits such as neuroticism and agreeableness [46]. 
Items were coded as patient-level factors if they assess 
constructs that represent aspects of the individual 
patients who will receive the innovation directly or indir- 
ectly. These aspects could include patient characteristics 
such as ethnicity or socioeconomic status (Barriers and 
Facilitators Assessment Instrument [47]), and patient 
needs {e.g., 'the proposed practice changes or guideline 
implementation take into consideration the needs and 
preferences of patients'; ORCA [27]). Finally, items 
were coded as innovation-level factors if they assess 
constructs that represent aspects of the innovation that 
is being implemented. These aspects could include the 
relative advantage of an innovation above existing 
practices [26] and quality of evidence supporting the 
innovation's efficacy (ORCA [27]). 

The coding process was item-focused rather than 
construct-focused, meaning that each item was evaluated 
individually and coded as representing a construct reflecting 
a structural, organizational, individual provider, individual 
patient, or innovation-level factor. In order for a meas- 
ure to be coded as representing one of the five factors, it 
had to include two or more items assessing a construct 
subsumed within the higher-order factor. We chose an 
item-focused coding approach because there is consider- 
able heterogeneity across disciplines and across researchers 
regarding psychometric criteria for scale development the 
procedures by which constructs are operationalized. 

It is also important to note that we coded items based 
on the subject or content of the item rather than based on 
the viewpoint of the respondent who completed the item. 
For example, a measure could include items in order to 
assess the general culture of a clinical care organization 
from two different perspectives— the perspective of the 
individual provider, or from the perspective of adminis- 
trators. Though these two perspectives might be construed 
to represent both provider and organizational-level factors, 
in our review, both were coded as organizational factors 
because the subject of the items' assessment is the orga- 
nization {i.e., its culture) regardless of who is providing 
the assessment. 

Measures were excluded because items did not assess 
any of the five factors {e.g., they instead measured an 
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implementation outcome such as fidelity [48]), were 
utilized only in articles examining a non-health-related 
innovation (e.g., end-user computing systems [49]), were 
unpublished or unobtainable (e.g., full measure available 
only in an unpublished manuscript that could not be 
obtained from corresponding author [50]), provided insuf- 
ficient information for review (e.g., multiple example items 
were not provided in original source article, nor was 
measure available from corresponding author [51]), were 
redundant with newer versions of a measure (e.g., Typ- 
ology Questionnaire redundant with Competing Values 
Framework [52]), or were only published in a foreign 
language (e.g., physician intention measure [53]). 

In addition, we coded the implementation outcome 
present in each search article that utilized one of the 
retained measures in order to determine the relative 
predictive utility, or criterion validity, of each of these 
measures [54]. In essence, we wanted to determine 
whether each measure was reliably associated with one 
or more implementation outcomes assessed in the 
articles included in our review. In order to do so, two 
coders reviewed all search articles and identified which 
of five possible implementation outcomes was assessed 
based on the typology provided by Proctor et al. [6]^: 
adoption, or the 'intention, initial decision, or action to 
try or employ an innovation or evidence-based practice'; 
fidelity, or 'the degree to which an intervention was 
implemented as it was prescribed in the original proto- 
col or as it was intended by the program developers'; 
implementation cost, or 'the cost impact of an imple- 
mentation effort'; penetration, or 'the integration of a 
practice within a service setting and its subsystems'; 
sustainability, 'the extent to which a newly implemented 
treatment is maintained or institutionalized within a 
service setting's ongoing, stable operations'; or no im- 
plementation outcomes. In addition, for those articles 
that assessed an implementation outcome, we also coded 
whether each included measure was demonstrated to be a 
statistically significant predictor of the implementation 
outcome. Together, these codes indicate the extent to 
which each measure has demonstrated criterion validity in 
relation to one or more implementation outcomes. 

Reliability 

Together, the first and third authors (SC and CB) independ- 
ently screened study records, reviewed full-text articles, 
identified measures within articles, coded measures, and 
assessed criterion validity. At each of these five stages of 
coding, inter-rater reliability was assessed by having each 
rater independently code a random sample representing 
25% of the full items [55]. Coding discrepancies were 
resolved through discussion and consultation with the 
second author (AD). 



Results 

Literature search results 

As depicted in Figure 2, these search strategies yielded 
589 unique peer-reviewed journal article records. Of 
those, 210 full-text articles were reviewed and vetted fur- 
ther, yielding a total of 125 full-text articles from which 
measures were extracted. A total of 112 measures were 
extracted from these retained articles. Across each of the 
stages of coding, inter-rater reliability was relatively high, 
ranging from 87 to 100% agreement. 

Our screening yielded a total of 62 measures. Table 1 
provides the full list of measures we obtained. (See 
Additional file 2 for a list of the names and citations of 
excluded measures.) For each measure, we provide infor- 
mation about its name and original validation source, 
whether it includes items that assess each of the five 
factors, information about the constructs measured, and 
the implementation context(s) in which the scale was 
used: healthcare (e.g., nursing utilization of evidence-based 
practice, guideline implementation), workplace, education, 
or mental health/substance abuse settings. In addition, we 
list information about the criterion validity [54] of each 
measure by examining the original validation source and 
each search article that utilized the scale, the type of im- 
plementation outcome that was assessed in each article, 
and whether there was evidence that the measure was 
statistically associated with the implementation outcome 
assessed. It is important to note that we utilized only the 
125 articles eligible for final review and the original valid- 
ation article (if not located within our search) in order to 
populate information for the criterion validity and imple- 
mentation context. Thus, this information represents only 
information available through these 125 articles and not 
from an exhaustive search of each measure within the 
available empirical literature. 

Factors assessed 

Of the 62 measures we obtained, most (42; 67.7%) 
assessed only one type of factor. Only one measure— the 
Barriers and Facilitators Assessment Instrument [47] — 
included items designed to assess each of the five factors 
examined in our review. 

Of the five factors coded in our review, individual pro- 
vider and organizational factors were the constructs 
most frequently assessed by these measures. Thirty-five 
(56.5%) measures assessed provider-level constructs, 
such as research-related attitudes and skills [56-58], per- 
sonality characteristics (e.g.. Big 5 Personality [46], and 
self-efficacy [59]). 

Thirty-seven (59.7%) measures assessed organizational- 
level constructs. Aspects of organizational culture and 
climate were assessed frequently [45,60] as were measures 
of organizational support or 'buy in' for implementation of 
the innovation [61-63]. 
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125 Full-text articles 
assessed for measures 





1 12 Measures identified and 
retained for 5-factor coding 



62 Measures containing 
items representing 1 or 
more of 5 factors 



50 Measures excluded 

18 No 5-factor constructs 

12 Unpublished or unobtainable 
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6 Redundant with existing 
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5 Insufficient information 
2 Not available in English 



Figure 2 Systematic literature review process. 



Innovation-level constructs were measured by one 
quarter of these measures (16; 25.8%). Many of these 
measures assessed constructs outlined in Roger's diffusion 
of innovations theory [18] such as relative advantage, com- 
patibility, complexity, trialabUity, and observability [26] . 

Structural-level (5; 8.1%) and patient-level (5; 8.1%) 
constructs were the least likely to be assessed. For ex- 
ample, the Barriers and Facilitators Assessment Instru- 
ment [47] assesses constructs associated with each of 
the five factors, including structural factors such as the 
social, political, societal context and patient factors such 
as patient characteristics. The ORCA [27] assesses pa- 
tient constructs in terms of the degree to which patient 
preferences are addressed in the available evidence 
supporting an innovation. 



Implementation context and criterion validity 

Consistent with our search strategies, most (47; 75.8%) 
measures were developed and/or implemented in 
healthcare-related settings. Most measures were utilized 
to examine factors that facilitate or inhibit adoption of 
evidence-based clinical care guidelines [56,64,65]. How- 
ever, several measures were utilized to evaluate imple- 
mentation of health-related innovations in educational 
{e.g., implementation of a preventive intervention in 
elementary schools [66]), mental health (technology trans- 
fer in substance abuse treatment centers [45]), workplace 
{e.g., willingness to implement worksite health promotion 
programs [67]), or other settings. 

Surprisingly, almost one-half (30; 48.4%) of the measures 
located in our search neither assessed criterion validity in 
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Alberta Context Tool (ACT) 
[86] 



Appraisal of Guidelines, 
Research, and Evaluation in 
Europe (AGREE) scale [91] 

Attitudes, Perceived 
Demand, and Perceived 
Support (ARTAS) [93] 



Barriers and Facilitators 
Assessment Instrument [47] 



Barriers to Implementation 
of Behavioral Therapy (BIBT) 
[95] 



Barriers to Research 
Utilization Scale (BARRIERS) 
[56] 



0: culture, leadership, evaluation, social capital, 
informal interactions, formal interactions, structural 
and electronic resources, organizational slack 



I: Scope and purpose, stakeholder involvement, rigor 
of development, clarity and presentation, applicability, 
editorial independence 

S: Funding and policy support 

0: Management support 

PA: Patient benefit 

I: Adaptability and feasibility 

S: Social, political, societal context 

0: Organizational context 

PR: Care provider characteristics 

PA: Patient characteristics 

I: Innovation characteristics 

0: Institutional constraints, insufficient collegial 
support 

PR: Philosophical opposition to evidence-based 
practice 

PA: Client dissatisfaction 

0: Setting barriers and limitations 

PR: Research skills, values, and awareness of EBP 

I: Quality and presentation of research 



[86] 


Adoption* 


Healthcare 


[87] 


Adoption* 




[88] 


None 




[89] 


None 




[90] 


None 




[91] 


None 


EBP government 


[92] 


None 


support 




organizations 


[93] 


Adoption* 


Healthcare 


[47] 


Adoption* 


Healthcare 



[94] 



[95] 
[96] 
[97] 



[56] 
[77] 
[98] 
[99] 
[100] 
[101] 
[102] 
[103] 
[78] 
[79] 



None 



None 
None 
None 



None 

Adoption* 

None 

None 

None 

None 

None 

None 

Adoption 

None 



IVIental Health/ 
Substance Abuse 



Healthcare 



n 

CD 

o 



Table 1 Coded measures [N = 62) (Continued) 



Big 5 Personality (e.g., NEO- 
FFI) [46] 



California Critical Tliinking 
Dispositions Inventory [106] 



Clinical Practice Guidelines 
Implementation Instrument 
[64] 

Community-Level Predictors 
[109] 

Competing Values 
Framework [62] Adapted 
from [110,111] 

Context Assessment Index 
[112] 

Coping Style: Supervisory 
Working Alliance Inventory 
[113] 

Decision-Maker Information 
Needs and Preferences 
Survey [114] 



Dimensions of the Learning 
Organization Questionnaire 
[115] 



Edmonton Research 
Orientation Survey (EROS) 
[58] 

Electronic Health Record 
Nurse Satisfaction Survey 
(EHRNS) [117] 



PR: Personality attributes (openness, 
conscientiousness, extraversion, agreeableness, 
neuroticism) 

PR: Inquisitiveness, systematicity, analyticity, truth- 
seeking, open-mindedness, self-confidence, maturity 



0: Context features 
: Evidence 

S: Poverty and population 

0: Organizational culture (hierarchical, entrepreneurial, 
team, and rational) 

0: Collaborative practice, evidence-informed practice, 
respect for persons, practice boundaries, evaluation 

PR: Coping style 



S: Financial resources, impact of regulations and 
legislation 

0: Support for EBP 

PR: Preferences for EBP information 

: Quality, relevance of EBP 

0: Continuous learning, inquiry and dialogue, 
collaboration and team learning, systems to capture 
learning, empower people, connect the organization, 
provide strategic leadership for learning, financial 
performance, knowledge performance 

PR: Valuing research, research involvement, being at 
the leading i 



Satisfaction with innovation 



[80] Adoption* 
[104] None 
[105] None 



[46] 
[66] 



None 
Fidelity* 

[106] None 



[107] Adoption* 
[108] None 
[64] None 



[113] 
[66] 



[58] 
[79] 
[104] 
[117] 



Education 



Healthcare 



Healthcare 



[109] Adoption* Healthcare 
[62] None Healthcare 



[112] None 



None 
Fidelity* 



[114] None 



Healthcare 
Education 

Healthcare 



[115] None Healthcare 
[116] Adoption* 



None 
None 
None 
None 



Healthcare 



Healthcare 
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Table 1 Coded measures {N = 62) (Continued) 



EPC [52,1 



Evidence Based Practice 
Attitude Scale [24] 



Evidence-Based Practice 
Beliefs Scale [57] 



Evidence-Based Practice 
Questionnaire [126] 



Facilitators Scale [105] 



Four As Research Utilization 
Survey [127] 

General Practitioners' 
Perceptions of the Route of 
Evidence-Based Medicine 
[128] 

Group Cohesion Scale [130] 



Guideline Implementability 
Appraisal (GLIA) [132] 

Healthy Heart Kit 
Questionnaire [26] 



nformation System 
Evaluation Tool [134] 

Intention to Leave Scale 
[135] 



PR: Cognitive response style 



PR: Intuitive appeal of EBP, openness to new practices 



PR: Attitudes about EBP 



PR: EBP attitudes, knowledge, and skills 



Q: Support for research 
PR: Education 

: Improving utility of research 

Q: Organizational barriers to access, assess, adapt, and 
apply EBP 

Q: Organization support for EBP 
PR: Attitudes toward EBP 
I: Quality of evidence 

0: Perceived group attractiveness and cohesiveness 

: Implementability 

0: Type of practice 

PR: Perceived confidence and control 

: Relative advantage, compatibility, complexity, 
trialability, observability 

I: Usability and usefulness of innovation 
0: Work rewards, people at work, work load 



[52] 
[118] 
[119] 
[24] 
[120] 
[121] 
[122] 
[123] 
[57] 
[124] 
[122] 
[125] 
[126] 
[80] 
[77] 
[105] 
[98] 
[101] 
[127] 

[128] 
[129] 

[130] 
[131] 
[132] 
[133] 
[26] 



Mental Health/ 
Substance Abuse 



None Healthcare 
Adoption* 
None 
None 
Adoption 
None 
Adoption 
None * 

Adoption* Healthcare 
None 
None 
None 
Adoption* 
Adoption* 
Adoption* 
None 
None 
None 
Adoption 

None 
None 



Healthcare 



Healthcare 



Mental Health/ 
Substance Abuse 

Healthcare 



None Healthcare 
None 

None Healthcare 
None 

Adoption* Healthcare 



[134] None 



[135] None 



Healthcare 



Healthcare 
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Table 1 Coded measures {N = 62) (Continued) 



Job Satisfaction [1 36] 



Knowledge, Attitudes, and 
Expectations of Web- 
Assisted Tobacco 
Interventions [137] 

Knowledge Transfer 
Inventory (Personal 
Knowledge Transfer 
subscale) [138] 

Knowledge Transfer and 
Exchange Correlates [140] 



Leader IVlember Exchange 
Scale [141] 

Nurses Retention Index 
[142] 

Nursing Research Utilization 
Survey [143] 

Nursing Work Index [144] 



Organization Readiness to 
Change Assessment (ORCA) 
[27] 



Organizational Culture and 
Readiness for System-Wide 
Implementation of EBP 
(OCRSIEP) [147] 

Organizational Culture 
Survey [60,149] 



Organizational Learning 
Survey (OLS) [150] 



PR: Intention to leave nursing profession 

0: Communication and decision-making 

PR: Pay, fringe benefits, and promotion, close friends 
at work 

I: Knowledge, expectations, actions, networking, 
information seeking related to innovation 



PR: Knowledge acquisition and sharing 



S: Policymakers use of EBP, and funding support 
0: Communication and decision-making 
PR: Research skills, and research activities 
0: Leadership style, work environment 

PR: Intention to stay in nursing profession 

PR: Attitudes towards nursing research 
0: hospital characteristics 



0: culture, leadership, measurement, readiness for 
change, resources, characteristics, role 

PA: Evidence: Patient preferences 

I: Evidence: Disagreement, evidence, clinical 
experience 

0: organizational culture, readiness for system-wide 
integration of EBP 



0: Constructive Culture (motivation, individualism, 
support). Passive Defensive Culture (consensus, 
conformity, subservience) 

0: Clarity of purpose, leadership, experimentation and 
rewards, transfer of knowledge, teamwork 



[131] 
[136] 
[131] 



[138] 
[139] 



[141] 
[139] 
[142] 
[131] 
[143] 

[144] 
[83] 
[145] 
[27] 



[147] 
[148] 
[131] 
[149] 
[60] 
[66] 
[150] 
[145] 



None 
None 
None 



[137] None 



None 
None 



[146] Adoption* 



None 

Adoption* 

None 

None 

None 

Fidelity* 

None 

None 



Healthcare 



Healthcare 



Healthcare 



[140] Adoption Healthcare 



None Healthcare 
None 

None Healthcare 
None 

Adoption* Healthcare 

None Healthcare 

None 

None 

None Healthcare 



Healthcare 

Education 

Healthcare 
Workplace 



Table 1 Coded measures {N = 62) (Continued) 



Organizational Readiness for X X 

CInange [45] 



Organizational Social X X 

Contextt [1 52] 

Organizational/Psychological X 
Climate [60,153] 



Ottawa Acceptability of 
Decision Rules Instrument 
(OADRI) [154] 

Perceived Importance of 
Dissemination Activities 
[155] 

Pre-lmplementatlon 
Expectancies [59] 



Ouality Improvement X 

Implementation Survey 

[156] 

Rational Experiential X 
Inventory [158] 

Researcfi Conduct and X 
Research Utilization by 
Nurses Questionnaire [160] 

Research Knowledge, X 
Attitudes and Practices of 
Research Survey [161] 

Research Utilization X 
Ouestionnaire (RUQ) [162] 

Research Utilization X X 

Ouestionnaire (RUO) [65] 



0: Institutional resources, Organizational climate, 
motivational readiness 

PR: Staff personality attributes 
0: Climate, culture 
PR: Work attitudes 

PR: Job satisfaction, depersonalization, emotiona 
exhaustion, role conflict 



I: Acceptability of clinical practice guidelines 



[45] 
[78] 
[151] 
[152] 
[121] 
[60] 
[153] 
[66] 
[151] 
[154] 



None 

Adoption 

Adoption* 

None 

None 

None 

None 

Fidelity* 

Adoption* 

Adoption* 



Mental Health/ 
Substance Abuse 



Mental Health/ 
Substance Abuse 

Education 



Mental Health/ 
Substance Abuse 

Healthcare 



PR: Perceived Importance of dissemination activities 



[155] None 



University Health 
Researchers 



0: Teacher morale, leadership encouragement 



PR: Enthusiasm for Implementation, preparedness to 
implement, implementation self-efflcacy 

I: Compatibility, beliefs about the program 

0: Culture, leadership, information and analysis, 
strategic planning quality, human resource utilization, 
quality management, quality results, customer 
satisfaction 

PR: Rational and experiential thinking styles 



PR: Knowledge base for research, attitude towards 
research utilization 



[59] 



[66] 



[156] 
[157] 

[158] 
[159] 
[160] 



Adoption* 

Fidelity* 

Fidelity* 

None 
None 

None 

Adoption* 

None 



Education 



Healthcare 



Healthcare 



Healthcare 



PR: Research knowledge, attitudes, practice 



PR: perceived difficulty of research utilization activities, 
attitudes regarding utilization 

0: availability and support 

PR: attitude 



[161] 
[79] 
[104] 
[162] 

[65] 
[163] 
[108] 



None 
None 
Adoption* 
None 

None 
None 
Adoption* 



Healthcare 

Healthcare 
Healthcare 



Table 1 Coded measures (N = 62) (Continued) 



San Francisco Treatment 
Research Center Course 
Evaluation [165] 



Team Clied<-Up Tool [167] 



Team Climate Inventory 
[168] 

Team Functioning Survey 
[169] 

Team Organization and 
Support Conditions 
Questionnaire [61] 

Thieoretical-Domains 
Framework [1 70] 



Thieory of Planned Behavior 
Constructs (i.e., attitudes, 
norms, perceived behaviora 
control) [25] 



Therapist Characteristics and 
Barriers to Implementation 
[174] 

Worksite Health Promotion 
Capacity Instrument 
(WHPCI) - Health Promotion 
Willingness subscale [57] 



0: Organizational barriers to adopting EBP, agency 

Management, strategies to support EBP 

PR: Stage of change for using EBPs, attitudes 
regarding EBP, 

past experience with EBP 

0: Leadership, shared decision-making, shared vision 



0: Shared vision, shared decision-making, support, 
information sharing 

0: Team skill, support, and work environment 



0: organizational support, team organization, external 
change agent support 



0: IVlanagement support, organizational support and 
resources 

PR: Perceived knowledge, skills, and abilities, 
motivation 

PA: patient interest in treatment 

PR: Attitudes, norms, perceived behavioral control 



0: clinic type, location, organizational support 

PR: Perceived skills and ability, counseling discipline, 
level of education, workload, motivation 

0: Health promotion willingness 



[164] 
[165] 
[166] 



[25] 
[171] 
[38] 
[172] 
[173] 
[174] 



None 
None 
None 



Mental Health/ 
Substance Abuse 



[167] None Healthcare 

[83] Adoption* * 

[168] Adoption* Healthcare 

[123] None* 

[169] Adoption* Healthcare 

[83] None 

[61] None Healthcare 

[170] None Healthcare 



None 

Adoption* 

Adoption* 

Adoption 

Adoption* 

None 



Healthcare 



Mental Health/ 
Substance Abuse 



[67] Adoption* Workplace 



■D - 

m 3 
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g' S 



Notes. EBP = evidence based practice. *IVlea5ure was a statistically significant predictor of the implementation outcome noted, t This scale includes the organizational culture and organizational climate scales also 
developed by the same author (Glisson & James, 2002). ^Measure used to predict a patient health outcome. 
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their original validation studies nor in the additional 
articles located in our search. That is, for the majority of 
these measures, implementation outcomes such as adop- 
tion or fidelity were not assessed in combination with the 
measure in order to determine whether the measure is re- 
liably associated with an implementation outcome. Of the 
32 measures for which criterion validity was examined, 
adoption was the most frequent (29 of 32; 90.1%) imple- 
mentation outcome examined. Only a small proportion of 
studies (5 of 32; 15.6%) examined fidelity, and no studies 
examined implementation cost, penetration, or sustain- 
ability. Again, it is important to note that we did not 
conduct an exhaustive search of each measure to locate 
all studies that have utilized it in past research, so it is 
possible that the criterion validity of these measures 
has, in fact, been assessed in other studies that were not 
located in our review. In addition, it is important to 
keep in mind that the criterion validity of recently 
developed scales may be weak solely because these 
measures have been evaluated less frequently than more 
established measures. 

Discussion 

Existing gaps in measurement present a formidable barrier 
to efforts to advance implementation science [68,69]. In 
the current review, we addressed these existing gaps by 
identifying a comprehensive, five-factor multi-level frame- 
work that builds upon converging evidence from multiple 
previous frameworks. We then conducted a systematic 
review in order to identify 62 available measures that 
can be utilized to assess constructs representing structural-, 
organizational-, provider-, patient-, and innovation-level 
factors— factors that are each hypothesized to affect im- 
plementation outcomes. Further, we evaluated the cri- 
terion validity of each measure in order to determine 
the degree to which each measure has, indeed, predicted 
implementation outcomes such as adoption and fidelity. 
In total, the current review advances understanding of 
the conceptual factors and observable constructs that 
impact implementation outcomes. In doing so, it provides 
as useful tool to aid researchers as they determine which 
of five types of factors to examine and which measures to 
utilize in order to assess constructs within each of these 
factors (see Table 1). 

Available measures 

In addition to providing a practical tool to aid in research 
design, our review highlights several important aspects 
about the current state of measurement in implementation 
science. While there is a relative preponderance of measures 
assessing organizational-, provider-, and innovation-level 
constructs, there are relatively few measures available to 
assess structural- and patient-level constructs. Structural- 
level constructs such as political norms, policies, and 



relative resources/socioeconomic status can be important 
macro-level determinants of implementation outcomes 
[9,19,20]. Why, then, did our search yield so few available 
measures of structural-level constructs? Structural-level 
constructs are among the least likely to be assessed be- 
cause their measurement poses unique methodological 
challenges for researchers. In order to ensure enough stat- 
istical power to test the effect of structural-level constructs 
on implementation outcomes, researchers must typically 
utilize exceptionally large samples that are typically diffi- 
cult to obtain [17]. Alternatively, when structural-level 
constructs are deemed to be important determinants of 
implementation outcomes, researchers conducting im- 
plementation trials may simply opt to control for these 
factors in their study designs by stratifying or matching 
organizations on these characteristics [70] rather than 
measuring them. Though the influence of some structural- 
level constructs might be captured through formative 
evaluation [71], many structural-level constructs such 
as relative socioeconomic resources and population 
density can be assessed with standardized population- 
type measures such as those assessed in national surveys 
such as the General Social Survey [72]. 

Though patient-level constructs may be somewhat easier 
to assess, there is also a relative dearth of measures 
designed to assess these constructs. Though we might 
assume that most innovations have been tested for pa- 
tient feasibility in prior stages of research or formative 
evaluation [71], this is not always a certainty. Thus, 
measures that assess the degree to which innovations 
are appropriate and feasible with the patient population 
of interest are especially important. Beyond feasibility, 
other important patient characteristics such as health 
literacy may also affect implementation, making it more 
likely that an innovation will be effectively implemented 
with some types of patients but not others [3]. Measures 
that assess these and other patient-level constructs will 
also be useful in strengthening these existing measure- 
ment gaps. 

In addition to locating those measures outlined in 
Table 1, the current review also highlights additional 
strategies that will allow researchers to further expand 
the available pool of measures. Though measures utilized 
in research examining non-health related innovations 
were excluded from the current review, many of these 
measures (see Additional file 2) — and those identified in 
other systematic reviews [38,40] — could also be useful to 
researchers to the extent that they are psychometrically 
sound and can be adapted to contexts of interest. Further, 
adaption of psychometrically sound measures from other 
literatures assessing organizational-level constructs {e.g., 
culture [73,74]), provider-level constructs {e.g., psycho- 
logical predictors of behavior change [75,76]), or others 
could also offer fruitful measurement strategies. 
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Criterion validity 

Because of its primary relevance in the current review, 
we evaluated the criterion validity of the identified 
measures. Our review concluded that for a vast majority 
of measures, criterion validity has either not been 
examined or has not been supported. For example, al- 
though the BARRIERS scale was the most frequently 
utilized measure of those included in our review (i.e., 
utilized in 12 articles), only three of those articles 
utilized the measure to predict an implementation out- 
come (i.e., adoption [77-80]). Instead, this measure was 
used to describe the setting as either amenable or not 
amenable to implementation, though no implementation 
activity was assessed by the measure itself Thus, there is 
a preponderance of scales that currently serve descrip- 
tive purposes only. Though descriptive information 
obtained through these measures is useful for elicitation 
efforts, these measures might also provide important in- 
formation if they are used as predictors of implementa- 
tion outcomes. 

The lack of criterion validity associated with the ma- 
jority of the measures identified in the current review 
contributes to growing evidence regarding the weak 
psychometric properties of many available implementa- 
tion science measures. As Squires et al. [12] recently 
discussed in their review of measures assessing research 
utilization, a large majority of these existing measures 
demonstrate weak psychometric properties. Basic psycho- 
metric properties — reliability (e.g., internal reliability, test- 
retest reliability) and validity (e.g., construct validity, criter- 
ion validity) — of any measure should always be evaluated 
prior to including the measure in research [54]. This is 
especially true in the area of implementation science 
measurement, given that it is a relatively new area of study 
and newly developed measures may have had limited 
use. Thus, our review highlights the need for continued 
development and refinement of psychometrically sound 
measures for use in implementation science settings. 

Our examination of implementation outcomes also 
provides us with a unique opportunity to identify trends 
in the type and frequency of implementation outcomes 
used within this sample of the literature. Measures iden- 
tified in the current review have been predominantly 
developed and tested in relation to implementation 
outcomes that occur early in the implementation process 
(i.e., adoption, fidelity) rather than those outcomes that 
occur later in the implementation process (i.e., sustain- 
ability [81,82]). Presumably, our use of broad search 
terms such as 'implementation' or 'translational research' 
would have identified measures that have been utilized 
by researchers examining both early (e.g., adoption, fidel- 
ity) and later (e.g., sustainability) stages of implementa- 
tion. To that extent, our findings mirror the progression 
of the field of implementation science as a whole, with 



early theorizing and research focusing predominantly 
on the initial adoption of an innovation and more recent 
investigations giving greater attention to the long-term 
sustainability of the innovation. Future research will 
benefit by examining the degree to which the measures 
identified in our search (and the constructs they assess) 
affect later-stage outcomes such as penetration and sus- 
tainability in implementation trials or affect patient health 
and implementation outcomes simultaneously in hybrid 
effectiveness-implementation designs [33]. In addition, 
though we did not include patient health outcomes in our 
criterion validity coding scheme, we did come across a 
number of articles that included these outcomes (e.g., 
number of bloodstream infections [83]) alone or in com- 
bination with other implementation outcomes. The use of 
these outcome measures underscores the notion that 
patient-level variables continue to be relevant in imple- 
mentation trials as they are in efficacy trials. 

Limitations and future directions 

The current review advances implementation science 
measurement by identifying a comprehensive, multi- 
level conceptual framework that articulates factors that 
predict implementation outcomes and provides a sys- 
tematic review of quantitative measures available to as- 
sess constructs representing these factors. It is important 
to note, however, that the specific search strategies 
adopted in this systematic review affect the types of 
articles located and the conclusions that can be drawn 
from them in important ways. 

Our use of multiple databases (i.e., MEDLINE, CINAHL, 
PsycINFO) which span multiple disciplines (e.g., medicine, 
public health, nursing, psychology) and search of the 
Implementation Science journal provides a broad cross- 
section of the empirical literature examining implemen- 
tation of health-related innovations. Despite this broad 
search, it is possible that additional relevant literature and 
measures could be identified in future reviews through the 
use of other databases such as ERIC or Business Source 
Complete, which may catalogue additional health- and 
non-health implementation research from other disciplines 
such as education and business, respectively. Indeed, a re- 
cent review of a wide array of organizational literatures 
yielded 30 measures of organizational-level constructs [40], 
only 13% of which overlapped with the current review. 
Thus, additional reviews that draw on non-health-related 
literatures will help to identify additional potentially 
relevant measures. 

Further, the specific keywords and database restrictions 
used to search for these keywords also impact the range of 
articles identified in the current review. For example, our 
use of the keyword 'health' was designed to provide a gen- 
eral cross-section of measures that would be relevant to 
researchers examining health-related innovations, broadly 
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construed. Its use may have omitted articles that utihzed 
only a specific health discipline keyword (e.g., cancer, 
diabetes, HIV/ AIDS) but not 'health' as a keyword in the 
title or abstract. Similarly, our measure-related keywords 
or thirteen expert-identified implementation science 
keywords [34] may have captured a majority, but not a 
complete, range of implementation science articles. We 
did not account for truncation or spelling variants in 
our search and, in two databases, we limited our search 
to two instrument-related fields that could have also 
resulted in missing potentially relevant studies and 
measures. Additional systematic reviews that utilize 
expanded sets of keywords (e.g., use of keywords as 
medical subject headings [MeSH]) may yield additional 
measures to complement those identified in the current 
review. Identification of qualitative measures would also 
further complement the current review. 

Finally, as noted earlier, assessment of criterion validity 
was based only on articles that were identified in the 
current search. Thus, because separate systematic searches 
were not conducted on each of the 62 individual 
measures, our assessment of criterion validity is based 
on a sampling of the literature available from the search 
strategy adopted herein. As a consequence, further re- 
search would be required in order to ensure an exhaust- 
ive assessment of criterion validity for some or all of the 
identified measures. 

Conclusion 

As the nexus between research and practice, the field of 
implementation science plays a critical role in advancing 
human health. Though it has made tremendous strides in 
its relatively short life span, the field also continues to face 
formidable challenges in refining the conceptualization and 
measurement of the factors that affect implementation suc- 
cess [84]. The current research addresses these challenges 
by outlining a comprehensive conceptual framework and 
the associated measures available for use by implementa- 
tion researchers. By helping researchers gain greater clarity 
regarding the conceptual factors and measured variables 
that impact implementation success, the current review 
may also contribute towards future efforts to translate these 
frameworks into theories. As has been demonstrated in 
nearly every domain of health, theory-based research — 
in which researchers derive testable hypotheses from 
theories or frameworks that provide a system of predict- 
able relationships between constructs — has stimulated 
many of the greatest advances in effective disease pre- 
vention and health promotion efforts [85]. So, too, must 
implementation science translate existing frameworks into 
theories that can gain greater specificity in predicting the 
interrelations among the factors that impact implementa- 
tion success and, ultimately, improve human health. 



Endnotes 

^It is important to note that we do not consider patient 
perceptions of the innovation as patient-level factors [28] 
because the object of focus is the innovation rather 
than the patient per se. Thus, patient perceptions of 
the innovation — similar to provider perceptions of the 
innovation — are considered to be innovation-level factors. 

^Given that Proctors [6] conceptualization of the con- 
structs of acceptability, appropriateness, and feasibility are 
redundant with our conceptualization of innovation-level 
factors, we omitted these three constructs from our coding 
of implementation outcomes. 
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Additional file 2: Excluded scales and reason for exclusion. 
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